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EXPERT AND NOVICE TEACHERS* ABILITY 
TO JUDGE STUDENT UNDERSTANDING 



Ellen Stader 
Terry Colyar 
David C. Berliner 

INTRODUCTION 

The ability to decode students* nonverbal behavior, particularly that which 
communicates a lack of comprehension, is an important skill for teachers to 
develop. At times students may be confused or totally lack understanding of 
some point and they may be unable or unwilling to ask teachers for help. From 
observations of the students* body language and facial expressions, the perceptive 
teacher decides whetlier there is a need to check on comprehension, provide more 
instruction or a different kind of instruction, assign more practice, and so forth. It 
is generally believed that experienced or expert teachers possess the ability to 
accurately assess their students comprehension, though research on this topic is 
not extensive. To study this topic knowledge is needed about how students know 
whether they comprehend something and how teachers at various stages of 
development read the cues emitted by students. This study is an attempt to learn 
more about these issues. 

Students* self-evaluation of comprehension. Some research on interpersonal 
communication in the classroom has examined students* ability to a^^ess their 
own comprehension. Bearison and Levey (1977) studied kindergarten, second, 
and fourth grade students' ability to judge their own understanding. After 
listening to questions, the students were asked to label each one as good or bad, 
based on their own ability to answer it given the information provided. In a 
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similar study (Ironsmith & Whitehurst, 1978), kindergarten, second, fourth and 
sixth grade students were exposed to an ambiguous or nonambiguous description 
presented auditcrally by the investigator, and then asked to identify a picture 
that was representative of the communication. In both studies, the ability to 
assess one's own comprehension increased with age. If we assume that these two 
studies measured the same construct among pre-school and grade school children, 
performance on a task requiring self -evaluation of knowledge may be strongly 
related to age. Both studies required an overt response to the investigator, and 
younger children in the samples may have been less willing to verbalize their 
confusion to an unfamiliar adult. The younger aged children may also lack the 
experience and vocabulary necessary to express their understanding of their own 
knowing. So, experience (number of years) in the educational system may be 
another factor affecting performance on a self-evaluation of knowledge task, 
when comparing across age groups. Some evidence of this is provided in 
Patterson, Cosgrove and 0*Brian (1980), who found that preschoolers were able to 
recognize noninformative messages as evidenced by their nonverbal behavior, 
including eye contact, body movement, hand movement and reaction time. But, 
the children were not adept at expressing their confusion verbally. From the 
extant research it would appear that the ability to emit non-verbal cues of 
comprehension/noncomprehension appears early, but that reliable verbal 
expression of this aspect of communicating is delayed until approximately the 
intermediate grade levels of school. 

Teachers' abilitv to decode ^-verbal cues. Jecker, Maccoby, Breitrose & 
Rose (1964) studied teachers* ability to decode both verbal and non-verbal cues 
emitted by students. Their work revealed that teachers are less perceptive at 
decoding nonverbal cues than verbal cues, and that their reliance on nonverbal 
cues can lead to confusion. Nonsignificant differences in decoding nonverbal cues 
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of comprehension were noted across three groups of subjects— new intern 
teachers, inexperienced teachers and experienced teachers. The investigators 
concluded that neither teacher training nor classroom experience are effective at 
improving this ability. In a follow-up study, Jecker, Maccoby & Breitrose (1965) 
determined that specialized training does improve accuracy in decoding student 
non-verbal cues of comprehension. Teachers in that study were provided with 
analysis forms prepared by the investigators, to use in evaluating student 
comprehension as seen in film clips of students attending to instruction. The 
teacher's attention was directed to a variety of factors (hand movement, body 
movement, blinking, etc.) which were judged to be the most salient features of the 
film clip being studied. Under these conditions, training was found to improve 
accuracy. The investigators also concluded that there are common patterns of 
non-verbal cues that can readily be identlHed. 

Machida (1986) completed a cross cultural study of Mexican American and 
Anglo teachers' ability to judge comprehension among anglo american, proficient 
bilingual and limited English speaking first graders. While slight cultural 
differences in nonverbal behavior across these groups were noted, teachers of 
both cultures were able to decode those cues accurately. No significant 
differences between teachers' groups were noted; however, the level of 
experience of the teachers was not examined. 

In the first study to be described novice, advanced beginner, and expert 
teachers were compared in their ability to judge student comprehension from 
non-verbal behavior. The non-vsrbal behavior was presented by a videotape 
showing students responding to a problem posed to them by a teacher. The 
students that were viewed on the tapes were carefully chosen based on their 
"feeling of knowing" (Metcalfe, 1986). That is, they were picked for the accuracy 
of their metacognitive thinking about how easy or difHcult a problem was to 
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solve. Children who possess well developed metacognitive strategies might be 
expected to provide clear non-verbal cues, that accurately reflect their 
comprehension and which contain the least am.biguity to observers. Under 
condiMons where students were judged to be capable of emitting accurate non- 
verbal cues, it was hypothesized that expert teachers would perform better than 
either advanced beginners or novice teachers, and that discussion of the criteria 
for their decisions would serve as training for higher performance on subsequent 
tapes. Two methods of discussion of tne cues emitted by the students constituted 
the independent variable. No prepared list of non-verbal cues for teachers to use 
as an aid to evaluation was provided. 

METHOD 

Design 

For study one, a 3 x 2 factorial design was used. The between subject 
factors were experience level of the teacher (Novice, Advanced Beginner, Expert) 
and training condition (Discussion vs. discussion with feedback). The dependent 
variables were accuracy in assessing student n^sponse (correct or error), and 
confidence in that decision (ratings from 1-5) 

Subjects 

9 novices, 10 advanced beginner, and 10 expert teachers were recruited 
from the Peoria UniHed School District in Arizona and Arizona State University 
teacher education classes. Expert teachers were nominated for inclusion in the 
study by their principals, assistant principals or staff development personnel, 
based on Berliner's (1988) criteria. Those designated as experts all had at least S 
years of recent consecutive teaching service. Advanced beginners were teachera 
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with one or two years teachii^g experience. Novices were students enrolled in 
teacher education course work, some of whom had student teaching experience, 
some of whom did not. 

Stimulus Tapes 

Peoria school district 4th grade students ir 3 classes were administered a 
ten item multiple-choice test presented auditorally by the students* regular 
classroom teacher, within the regular classroom setting, according to a 
prearranged script. Items on the test were designed to be at an appropriate 
listening level, and were at the third grade reading ?"'*el on the Fry Readability 
Chart. Questions were drawn from three areas: math reasoning, spatial 
relationships, and verbal reasoning. The questions each were designed to require 
two or three problem solving steps. (Example: John is bigger than Sam, Sam is 
bigger than Joe, is John bigger than Joe?) One repetition of e'u:h item was allowed. 
Only the response options were provided on the students* answer sheets. The 
questions were heard and had to be kept in memory. In addition to indicating 
their choice of response, students were instructed to rate their level of confidence 
in their responses on a scale of 1-5, with 1 indicating the lowest level of 
confidence. Sample items were used at the beginning of the test to give students 
practice with solving these kinds of problems and in assessing their level of 
conHdence. The tests were scored to identify students who had the highest 
proportion of high conHdence correct responses coupled with low conHdence 
incorrect responses. Pilot testing had revealed that students who answered in 
either way had the potential to exhibit the least ambiguous non-verbal cues 
because of the high degree of awareness of their own comprehension. A set of six 
target students were identified in this way. 



One week later students received a parallel form of the earlier test, while 
being videotaped In their regular classroom. In each classroom, two video 
camcorders were positioned in the front of the classroom, facing the students, one 
on either side. The students were told beforehand that the entire class would be 
on camera, though only the target students were actually being videotaped. Since 
natural responses were desired of the target students, seating of the target 
students was not altered to accommodate the fihning. Sixty individual clips of 
target students answering questions were obtained, each approximately 20-30 
seconds in length. The answer sheets of these students were scored to identify 
those items that were responded to correctly with the highest level of confldence 
or which were responded to incorrectly with the lowest level of confldence. Of 
the clips that flt this pattern, 30 were chosen at random, with consideration for 
obtaining a representative sample of each of the six target students. These clips 
were divided in half and transferred in random order to two videocassette tapes, 
providing two parallel forms of the stimulus materials. 

Procedures 

The 29 teachers were randomly assigned to one of Ave groups to view the 
tapes. With one exception, each viewing group contained at least 2 teachers from 
each level of teaching experience. Groups were assigned to one of two treatments, 
discussion or discussion with feedback. The experiment was conducted in a small 
clasrroom on the campus of an elementary school. Teachers were seated in a 
horseshoe arrangement facing a TV monitor, with a video camcorder to the left of 
the monitor recording their work. 

After a group had assembled, teachers were told they were going to view a 
tape, without sound, showing 4th grade <;tudents responding to a test. The 
subjects were informed that the test items were designed to elicit problem solving 
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behavior, that these students were chosen for their ability to accurately assess 
what they did and did not know; and that they were chosen because they might 
emit accurate non-verbal cues as to whether or not they had comprehended the 
question being asked. The importance of teachers' ability to assess students' 
comprehension using non-verbal cues was discussed. Non-verbal cues were 
referred to as facial expressions and body language, with no reference made to 
examples of either. The subjects were told to watch each clip and to determine 
from the students' non-verbal cues the accuracy of the student's response, and to 
rate their level of confidence in the judgment they had rendered. Subjects were 
informed that the clips were random selections, and that there was no guarantee 
that the clips were half correct and half incorrect responses. Subjects in the group 
responded individually on a printed answer sheet, rating each student as having 
responded correctly or incorrectly to each item, and rating their own level of 
confidence in their response. Following this, the group of subjects viewed the 
same tape for a second time. For those assigned to the discussion treatment 
group, after each clip, subjects were encouraged to discuss the criteria for their 
response with one another and to indicate which non-verbal behaviors were most 
salient in their decision. One of the first two researchers served as group 
moderator, asking questions such as: "Who thought this student got the item right 
(or vifrong)?" "Why did you think so?" " Anyone disagree?" "Anyone else want to 
comment?". These subjects were not provided feedback on the accuracy of their 
response. The same kind of discussion followed the viewing of each of the 15 
clips on this tape. After all the clips on tape one had been seen and discussed, the 
group viewed the 15 clips comprising tape 2, once again noting their response and 
their level of confidence in their response. Answer sheets were then collected, 
concluding the participation of the subjects in the study. For the feedback 
treatment, teachers discussed the criteria for their 15 responses during the 



second viewing of tape 1, as in the discussion treatment, but they were also given 
feedback on the accuracy of their response each clip before moving to the next 
clip. All groups were videotaped. The videotapes of the teachers' discussions 
were analyzed to inquire if qualitative differences appeared among the 
discussion groups, and among the experts, advanced beginners and novices that 
comprised each group. 

RESULTS AND DISCUSSION 

Quantitative Analysis. For the discussion plus feedback treatment, the level 
of performance in correctly identifying non-verbal cues of comprehension 
increased in all groups. Compared to tape one the gains on tape two for experts, 
advanced beginners and novices were +1.7, +1.8, +1.0, respectively. The 
corresponding mean certainty changes were -0.1, +0.2, -0.1, an inconsistent 
pattern with changes of little magnitude. AncvaU on both dependent measures 
revealed that these differences were not statistically significant. For the 
discussion treatment, performance between tape one and tape two increased for 
Experts (+0.05), but decreased for both advanced beginners and novices (-0.2, - 
2.3). The corre.«ponding mean certainty changes for experts, advanced beginners, 
and novices were +0.1, -0.9, +1.1, respectivdy. These gains were not statistically 
significant. As shown in Table 1, there is some support for our first hypothesis, 
that Experts would outperform both of the other groups of teachers; however, the 
differences between these groups were small. 

Qualitative Analysis. Analysis of the discussions held in conjunction with 
the viewing of the first tape provided useful information on the qualitative 
differences between the ways groups arrived at their decisions. It appeared as if 
experts in these groups tried to conteAtualize the student's behavior by 
repeatedly asking the moderator for information about the child and the tasK to 
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which they were responding-something very difficult to infer when no sound is 
provided on the tapes. They asked, also, how the student performed in school on 
a regular basis. They wanted to know about the testing environment, such as 
what was the point that the teacher stopped reading the question. They asked 
about the questions, and whether each child knew they were being singled out for 
taping. In each case, experts were told that this information was unavailable and 
they were to work only with the non-verbal cues that were presented. More than 
others, the experts seemed to feel that this was a near impossible task, and this 
belief may te reflected in the low levels of confidence they had to their own 
lesponses. The experts also reported relying on their knowledge of past students 
who behaved in a similar manner to arrive at a decision. They frequently were 
heard to make remarks like, '*0h, I've had a student like that**. It is interesting to 
note that experts consistently had a difficult time assessing the atypical student. 
One male student exhibited behaviors which experts interpreted as indicating low 
ability and low motivation (slumped over desk, head down and slow response 
time); however, subsequent to this experiment he was identified as gifted in 
intellectual potential and was recommended for placement in the district program 
for academically talented students. Obviously, his behavior indicated something 
much different. His responses to the problems he was given were consistently 
correct, and he had high conHdence in those responses. He may have felt the test 
was quite easy and behaved accordingly. 

Advanced beginners and novices appeared alike in that they did not 
contextualize their responses but personalized them. Each could be heard to make 
remarks such as, ''That's what I do when I don't understand'* or "I always flip u^y 
hair like her when I feel confident". Each novice and advanced beginner relied on 
their own patterns of behavior to evaluate the clips and invariably 
overgeneralized using words such as always and never. Results from the analysis 
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of tape 1 indicate that their decisions were reasonably accurate. Yet, the evidence 
from their performance with tape 2 suggests that they were easily swayed from 
their own opinions, especially in the discussion only treatment, when the validity 
of their own decision processes could not be verified. Their responses to tape 2 
were less accurate. 

Regardless of whether a teacher viewed the student as correct or incorrect, 
similar cues were noted. These cues have been sorted into 11 categories and are 
presented as Table 2. ^^hat is interesting to note is that teachers did not attribute 
the same meaning to the cue. For example, when discussing latency of response, 
which was a cue frequently used in making judgments of comprehension or 
noncomprehension, one teacher would report that the child had responded slowly 
and so was taking his time, concentrating on the problem, and probably reaching 
the correct answer. Another teacher, however, would identify the speed of the 
child*s response as an indicator of indecision and would judge the child to be 
inconect. This kind of lack of consensus about the meaning of the cues was noted 
in all three groups of teachers. Discussions where conrensus was lacking seemed 
to influence later decisions, especially among teachers in the discussion treatment, 
and particularly among the novice teachers in that treatment, whose performance 
fell off sharply on Tape 2. Presumably, the decrement they displayed was 
because they were listening to those with greater experience and status, and 
altering the decision rules they were using in their subsequent responses. 

Conclusion 

Six things were learned from this investigation. First, experts were not any 
better at this task than other teachers, as judged by performance on tape one. 
Nevertheless, expert teachers did, as hypothesized, outperform those with less 
experience on tape 2. But these differences were slight and not significant. 
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Second* it appeared as if there was a training effect. The tape two scores 
were higher for experts, advanced beginners and novices when they had feedback 
about their performance along with discussion of the particular film clip. The 
feedback about the correctness or incorrectness of their responses apparently 
served to modify their performance. 

Third, non-verbal cues in and of themselves are easily recognizable; 
however, the interpretation of them is not consistent. Different people utilize 
different social perspectives in their assessment of the meaning of behavior 
er\itted non-verbally. The perspective used by any given observer is genci 'V 
accurate; however, there are instances when cues emitted are inconsistent with 
the observer's conceptual framework and incorrect judgments are made. These 
appear to be relatively resistant to change. Confusion also arises when two or 
more people attempt to reconcile their differing judgments by combining cri'jria. 

Fourth, experts had greater needs to contextualize the situation— to seek 
more meaning and information--than did novices or advanced beginners. 

Fifth, experts made use of their past classroom experience, their case 
knowledge, to interpret the behaviors that they viewed. Novices and advanced 
beginners, on the other haud: relyed on their own personal knowledge, mvore often 
judging events based on what they did or on how they act. 

Final y, expert teachers develop composite pictures of the common and the 
usual and are as much at a loss as less experienced teachers when they encounter 
the atypical or unusual student. 



Because expert, experienced teachers seemed to be onJy marginally better 
at interpreting the comprehension cues emitted by students tlian those less 
experienced, it was hypothesized that their presumed ability might be confined to 
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their owa classes. The ability to comprehend these cues may require personal 
knowledge of the students being judged, and/or extensive time with them. The 
consistent questioning by the expei^ teachers in our sample, about the context for 
instruction and the background of the student, suggested that the ability to read 
comprehension cues may exist among experienced/expert teachers, but only in 
their own class. A way to gain some insight into that hypothesis was to show the 
tapes that had been created to the three 4th grade teachers of the target students. 
Two of the students in the clips were currently enrolled in each one of the classes 
of the three teachers, but the teachers had incidental knowledge of the other four 
target students from participation in field trips; playground, lunch and recess 
duty; knowledge of older siblings; and conversations with other teachers 
regarding their students throughout the year. Each of these tiree teachers had 
been identified by their principals as expert. 

Teachers viewed the tapes with the first researcher, individually, during 
regular school hours. After viewing tape 1 in its entirety, each teacher received 
feedback on the correctness of their responses and immediately viewed tape 2. 
The mean number correct for tape 1 was 10.6 with a confidence level of 2.8. 
Performance on tape 2 this score increased to 11.0, and confidence ratings had 
increased to 3.1. (See Table 3). These levels of performance and confidence were 
considerably higher than for the subjects in Study One. (See Table 1). Knowledge 
of the day to day behavior of the students apparently had a clear advantage. 
Consistently, teachers mediated their responses with ratings of the child's general 
ability e.g. ''He look unsure, but he's a good student, so I think he got it right" or 
other personal information e.g. "I don't know her but her sister was a good 
student so I think she got it right". When questioned whether the teachers we/e 
looking for non-verbal cues or simply using past performance as the criteria, each 
reported consideration of the cues but noted that pe::iOnal information about the 
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students was a factor in their final decision. The gifted child referred to earlier, 
who so often was misevaluated, was only evaluated correctly by his own teacher. 
Also, another child emerged as an atypical case. A female child was consistently 
rated by her teacher as being incorrect and unsure of herself when in fact this 
student had assessed her abilities well. Discussion with her teacher regarding 
this, after viewing both tapes, revealed that her teacher viewed her as an average 
student who would be more often wrong than right despite the fact that her 
grades were consistently at a C or better level. So, personal information used 
incorrectly can bias a teacher against a child as well as for them. 

In conclusion, past performance is a good predictor of future student 
performance for teachers, particularly when other information is ambiguous or 
unavailable. The combination of nonverbal cues and knowledge of the child 
probably allows for more accurate prediction than does either alone. But it is 
hard to sort out when teachers are acting as Baysian statisticians, computing 
present odds based on past performance, and when they are accurately decoding 
student non-verbal cues. 

Study 3 

It occurred to us that children may be better able than adults to decode the 
non-verbal cues of other children, since they depend on such cues to guide their 
everyday interpersonal behavior. So a further study was conducted. Twenty-two 
4th grade students in the Peoria School District were shown the two tapes in their 
regular classroom setting. Their teacher was present; however, she did not 
pfjt7cipate. Directions were simih" to those used for teachers but did not include 
information on the relevance of the study to teaching. Students .-emained seated 
at their desks and viute "right" or "wrong" on their answer sheet, a looseleaf 
sheet they had numbered to 15. Children viewed the first tape and some were 
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allowed to respond orally to each Him clip. Of those children that raised their 
hand, som^ were called on to give their decision and provide an explanation for 
that decision. Aiicr several students responded to each clip, there was a show of 
hands about whether the student in the video was right or wrong, and students 
were given feedback. As the clips continued, students showed the same confusion 
exhibited by teachers. They observed similar non-verbal cues, but int^^rpretation 
was often different. Also, the students who were subjects occasionally made 
remarks about the personal characteristics of the student being viewed. This 
apparj;ntly influenced their decision. Responses to tape 2 were covert with no 
discussion allowed. 

The mean number correct for tape 2 was 5.82, well below the scores of most 
respondents in Experiment I, but closest to the performance of novices in the 
discussion group. Many of the same comments stated earlier apply here. 
Students were best when they followed their own instincts. They seemed to 
become confused, and their performance deteriorated, when competing 
interpretations were offered. 

Study 4 

Because there appeared to be a training effect in study one, attributable to 
the feedback about the correctness or incorrectness of the subjects responses, 
further investigation seemed called for. Two preservice classes of teachers, 
novice teachers who had not yet participated in student teaching, were recruited 
to participate in this study. By random assignment one class was assigned to the 
discussion treatment, the other to the discussion plus feedback treatment. 
Procedures similar to those used in Study One were followed. In this study, 
however, we sought information about how experience with children in social 
situations, and self ratings of social intelligence, might affect performance. A brief 
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questionnaire was administered to these pre-service teachers assessing 
experience (and self reports of competency in understanding), with childi'en of 
the type presented on the video clips. A single score was derived from the 
questionnaire, indicating the degree of social sensitivity the subjects might have 
with students. 

Table 4 presents the performance data obtained in this study. As can be 
seen, there were no gains from tape one to tape two in the discussion treatment. 
On the other hand, in the discussion plus feedback condition, gains were made 
from tape one to tape two. The mean gain of .92 points was statistically 
significant (t = 2.29, P < .03). 

The correlations with the measure of social sensitivity were generally near 
zero, with one exception, (r = .55, between score on tape 1 and score on the 
measure of social sensitivity in the discussion plus feedback ^lass). But no trends 
could be detected. There appeared to be no relation between this measure of 
social sensitivity and a person's accuracy in reading non-verbal cues of 
comprehension in fourth graders. 

Overall Findings 

For the teacher education community, the major finding derived from these 
four small inquiries is that accuracy in decoding student comprehension from 
non-verbal cues is trainable. Study one provided some evidence of that, study 
four confirmed that finding. This conclusion was also reached by Jecker, Maccoby 
and Breitrose (1965). Our data suggest, however, that it was not just feedback 
that changed performance. The feedback about a particular child probably 
provided more information about that child's typical behavior, which aided a 
viewer in assessing subsequent clips of the child. The feedback provided a trace 
of the kind of information that the experts sought and that the teachers who were 
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familiar with the students used in their assessments. Although experts appeared 
only marginally better than others in this skill, when they also have knowledge 
about the students classroom performance, they appear to do particularly wsU at 
this task. Any training program or further research in this area must focus on 
the role that contextual knowledge plays in assessing the meaning of a child's 
non-verbal behavior. 



Table 1. Mean accuracy and confidence by treatment and tape 

Experts AdvMced Beginner Novice 



Ta<w> 1 Tmw> 2i 


IVape 1 Tane 2 


Taoe 1 Tane 2j 


rrape 1 Taue 2 


Tape 1 Tape 2 


n-ane 1 JEanc_2_^ 


Discussion 


7.5 8.0 


2.0 2.1 


8.0 2.9 


7.8 2.0 


8.8 6.5 


2.1 3.2 


Discussion 
plus feedbacl 


,7.8 9.5 


3.1 3.0 


7.6 9.4 


2.2 2.4 


8.2 9.2 


2.6 2.5 
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Table 2. Cues used in judgments of comprehension from non-verbal 
behavior emitted by students (combined for tapes one and two) 

Frequency of Mention 



Cup Eirnerts 


A.ayaTiCcu Dcginner 


iiitfyii.tr 




Quick/No hesitations 


21 


00 






Slow/Hesitant 


10 








Confident/Secure/ 
Relaxed/Satisfied/ 
Aggressive/Positive 


48 








Lack of confidence/ 

Insecure/Tense 

Puzzled/Unsure 


37 


35 


35 




Attends/Interested/Listens 


17 


5 


13 




Distracted/Bored/Apathetic 


28 


14 


12 




Deliberate/Reviews Work 


19 


16 


19 




Guesses/Second Thoughts 


12 


14 


9 




Eye Movements 


32 


37 


27 




Facial Expressions 


20 


15 


14 





Extraneous Movements 36 41 21 
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Table 3. Mean accuracy and confldence by teachers who knew the 
target students 



Tane One Tane Two 


Accuracy 


10.06 


11.00 


Confidence 


2.08 


3.01 
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Table 4. Mean accuracy score for novice teachers in two treatments 



Tane One 


1 ape — LssL 


Discussion 


8.45 


8.07 


(n = 29) 


(1.30) 


(1.44) 


Discussion 






plus 


8.67 


9.59 


feedback 


(1.88) 


(1.32) 


(n = 29) 







^ Figures in parenthesis are standard deviations 
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